The LLM section itself is for a specific model. These fallback scenarios may save typing, but end up potentially mixing values from different sections that imo shouldn't.
Non-present values in a custom llm section should only fall back to data class defaults, otherwise it is more complex, not easier, to find out where a value comes from.
E.g. a max_output_tokens value might be left out in "[llm.sonnet]" but suddenly get a value from the "[llm]" section which is just wrong as it is for a different model.
Then it should fallback to the default dataclass value.
Not to mention, that changing the model in the UI completely makes the toml file an unknown in which values are taken.

@tobitege You make good points, and I think there is a solution for the first: the case happens because you need max_out_tokens for the LLM in [llm], then you can name that LLM, right? e.g. [llm.llama]. Then they override as you wish, if I understand correctly: the attributes of llama with llama, claude with claude.

A model is optional in the [llm] section. We can use it or not, at all. As you say, if it helps to actually choose exactly what you want to choose: then name that group.

The issue with how the UI works with toml is tracked here. Just to note here:

Custom LLM configs were developed for CLI, not UI. I'm pretty sure they did not interfere with the UI at all until - as you pointed out recently in slack - the unexpected happened: setting one as "agent" LLM made it actively used with the UI too. That was, as far as I know, an unintended use case, and frankly I didn't even think it could happen until you mentioned it. I didn't even get to test it yet. This issue is not about that.

I think sorting it all out will not be trivial, but if I were to guess, probably the solutions for the interplay between UI and toml will be based on:

• have LLM settings (all of them or almost) in the UI. That should solve some issues, and, importantly,
• the suggestion in the linked topic, to actually save in the toml file when you override from the UI; and of course read it, not parts, not unintently, but read it all, should fix many of these things.

IMHO, the way it works now is mostly workable, except for that case - agent llm. I'm not sure how we should address that, without very significant changes.

The LLM section itself is for a specific model.

On further thought, I see your point better. Maybe the solution here doesn't require a lot, just a way to define a "default llm". That means to separate these two views on the [llm] section:

• when we use CLI with -l, the [llm] section means nothing as an llm, it's not used as such, so it's used as generic
• otherwise, [llm] section means the llm, the default llm to which the agent falls back to, and other circumstances that needs an llm for completion.

Actually I don't fully see how you encounter the first case you mention. I mean, if you are running main.py --llm_config sonnet sometimes and main.py without config another time, you can name the second one with a custom name. Then you will not get fallbacks.
If that's your preferred workflow, it seems like you can have it, this PR doesn't break the ability, it just restores the ability to switch with 20+ configs, basically. 😅

E.g. a max_output_tokens value might be left out in "[llm.sonnet]" but suddenly get a value from the "[llm]" section which is just wrong as it is for a different model.

Can you please tell how exactly you were encountering this case? e.g. how were you running.

@enyst is this something that will still be pursued?

I've come around to see the benefit to have the [llm] entries serve as defaults for any non-existant key in custom sections.
This indeed should make it easier for the user and shorten the toml setup.
But could we at least add corresponding documentation for this feature in this PR?

Will do, Tobi, it's on my todo list, I'll get to it tomorrow!

@openhands-agent Read this PR diff carefully and understand what it's doing. Read the *_config.py files to understand the config implementation we have. Read also config.template.toml, which is very important because it does already attempt to document how the configuration works.

Then, check out docs/modules/usage for Configuration Options, to see what we have documented so far. Add another .md file to document the behavior of this particular feature: named (custom) LLM configurations. Make sure it does include accurate description of the feature and how to use it.

Note: you should include a link to this new page, in the Configuration Options page under LLM Config, and specify that it is ONLY used with the development setup (development.md, you can find that too).

OpenHands started fixing the pr! You can monitor the progress here.

Overview of Changes:
✓ All issues appear to have been successfully resolved through:

• Creation of dedicated documentation for custom LLM configs
• Comprehensive coverage of feature details and usage
• Proper integration with existing documentation
• Clear labeling of development-only status

Status: Complete - No remaining issues identified. The implementation addresses all requested documentation needs while maintaining proper structure and transparency about feature limitations.

This is ready for review again, on similar principle as before, consistent fallbacks.

FWIW I'm changing my mind about a few of them, the auth-relevant ones. I doubt excluding some should be implemented on the current code design though, it needs a refactoring first IMO.

@enyst is it worth it to ping people for review here now?